Identification of page sharing opportunities within large pages

ABSTRACT

Memory performance in a computer system that implements large page mapping is improved even when memory is scarce by identifying page sharing opportunities within the large pages at the granularity of small pages and breaking up the large pages so that small pages within the large page can be freed up through page sharing. In addition, the number of small page sharing opportunities within the large pages can be used to estimate the total amount of memory that could be reclaimed through page sharing.

BACKGROUND

Operating systems and hypervisors that support execution of virtualmachines running in computer systems typically employ hierarchicallyarranged page tables to manage mapping of memory from a virtual memoryspace to a physical memory space that is divided into pages. When a pageis mapped from an entry of a page table at the lowest level, the size ofthe page that is mapped is typically 4 KB. When a page is mapped from anentry of a page table that is one level higher, the size of the pagethat is mapped is typically 2 MB. When memory is plentiful, significantimprovements in memory performance is observed by mapping to large(e.g., 2 MB) pages. However, as memory becomes scarce, memoryperformance of an operating system or a hypervisor employing large pagesdegrades quickly and becomes worse than when small (e.g., 4 KB) pagesare employed.

SUMMARY

One or more embodiments provide techniques to improve memory performancein a computer system that implements large page mapping. According toembodiments, page sharing opportunities at the granularity of smallpages are identified within pages that are mapped large and thisinformation is used in selecting large pages to break up when memorybecomes scarce so that small pages within the large page can be freed upthrough page sharing.

A method of reclaiming memory in a computer system according to anembodiment includes the steps of selecting a large page based on anumber of small page sharing opportunities identified therein, updatingmappings for the memory so that a mapping to the selected large page ischanged to mappings to small pages, where at least one of the smallpages is a shared small page, and marking one or more of the small pagesas free.

A method of determining an amount of memory that can be reclaimedthrough page sharing in a computer system, according to an embodiment,includes the steps of scanning each of the large pages and determiningfrom the scanning a number of small pages therein that can be shared,and determining an amount of memory that can be reclaimed based on atotal number of small pages that can be shared as determined from thescanning of all of the large pages.

A computer system according to an embodiment includes virtual machinesexecuted therein and a hypervisor configured to support execution of thevirtual machines, wherein the hypervisor is configured to map a virtualmemory space to a physical memory space of the computer system that islogically partitioned into large pages and small pages, and to determinea number of small pages within a given large page whose contents are thesame as another small page.

Further embodiments of the present invention include a non-transitorycomputer-readable storage medium comprising instructions that cause acomputer system to carry out one or more of the above methods as well asa computer system configured to carry out one or more of the abovemethods.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a virtualized computer system that isconfigured to identify page sharing opportunities within large pagesaccording to one or more embodiments.

FIGS. 2A and 2B are conceptual diagrams that illustrate mappings ofsmall and large pages from a virtual memory space to a physical memoryspace.

FIGS. 3A and 3B are conceptual diagrams that illustrate changes in themappings of small pages according to one or more embodiments.

FIG. 4 depicts a flow diagram that illustrates a method that includesthe steps of identifying page sharing opportunities, according to anembodiment.

FIG. 5 depicts a flow diagram that illustrates a method that includesthe steps of breaking up large pages to enable page sharing, accordingto an embodiment.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of a virtualized computer system that isconfigured to identify page sharing opportunities within large pagesaccording to one or more embodiments. Host computer system 100 may beconstructed on a desktop, laptop or server grade hardware platform 102,such as an x86 architecture platform. Hardware platform 102 includes oneor more central processing units (CPU) 103, host physical memory 104,and other standard hardware components such as network interfacecontrollers (not shown) that connect host computer system 100 to anetwork and one or more host bus adapters (not shown) that connect hostcomputer system 100 to a persistent storage device, illustrated hereinas storage system 160.

A hypervisor 114 is installed on top of hardware platform 102.Hypervisor 114 supports multiple virtual machine execution spaces 116₁-116 _(N), within each of which a VM process may be executed toinstantiate corresponding VMs 120 ₁-120 _(N). For each of VMs 120 ₁-120_(N), a resource scheduling module 149 of hypervisor 114, which includesa CPU scheduling module and a memory scheduling module, manages acorresponding virtual hardware platform (i.e., virtual hardwareplatforms 122 ₁-122 _(N)) that includes emulated hardware such asvirtual CPUs (vCPUs) and guest physical memory. Each virtual hardwareplatform 122 supports the installation of a guest operating system (OS)(e.g., guest OS 132). In each instance, the guest OS provides user-levelapplications running in the virtual machine, e.g., APPS 113, aninterface to the virtual hardware platform of the virtual machine.

It should be recognized that the various terms, layers andcategorizations used to describe the virtualization components in FIG. 1may be referred to differently without departing from theirfunctionality or the spirit or scope of the invention. For example,virtual hardware platforms 122 ₁-122 _(N) may be considered to be partof virtual machine monitors (VMM) 140 ₁-140 _(N) which implement thevirtual system support needed to coordinate operations betweenhypervisor 114 and their respective VMs. Alternatively, virtual hardwareplatforms 122 ₁-122 _(N) may also be considered to be separate (e.g., asa component of its corresponding virtual machine since such platformsinclude the hardware emulation components for the virtual machine) fromVMMs 140 ₁-140 _(N), and VMMs 140 ₁-140 _(N) may be considered to beseparate from hypervisor 114. One example of hypervisor 114 that may beused is included as a component of VMware's vSphere product, which iscommercially available from VMware, Inc. of Palo Alto, Calif. It shouldfurther be recognized that other virtualized computer systems arecontemplated, such as hosted virtual machine systems, where thehypervisor is implemented in conjunction with a host operating system.

CPU 103 has a memory management unit (MMU) 105 that carries out themappings from a virtual address space of VMs 120 ₁-120 _(N) or any otherapplications running on hypervisor 114 to a physical address space ofmemory 104 (referred to herein as the “host physical address space”)using either a translation look-aside buffer (not shown) or page tables(not shown) stored in memory 104. In embodiments using shadow pagetables to map guest virtual address spaces within VMs 120 ₁-120 _(N)directly to the physical address space of memory 104, the virtualaddress space of VMs 120 ₁-120 _(N) referred to herein corresponds toone of the guest virtual address spaces within VMs 120 ₁-120 _(N), andthe page tables referenced herein correspond to the shadow page tables.In embodiments using extended or nested page tables where guest virtualaddress spaces within VMs 120 ₁-120 _(N) are mapped to a guest physicaladdress space using guest pages tables and the guest physical addressspace is mapped to the host physical address space using extended ornested page tables, the virtual address space of VMs 120 ₁-120 _(N)referred to herein corresponds to the guest physical address space andthe page tables referenced herein correspond to the extended or nestedpage tables. The mappings may be to a small page (SP) 10 in memory 104or a large page (LP) 20 in memory 104. In the x86 architectures, thetypical size for small pages is 4 KB and the typical size for largepages is 2 MB. However, it should be recognized that embodiments may bepracticed with different small page sizes and different large pagesizes.

Although memory performance improves with the use of large pages, pagesharing at the granularity of large pages becomes more difficult forseveral reasons. First, finding large pages with matching contents isless likely. Second, large pages are more likely to be modified at whichpoint page sharing will have to be terminated. This is true even thoughsmall pages within the large page can still be shared. For this reason,embodiments provide a page sharing module 150 that is programmed toidentify opportunities for page sharing at the granularity of smallpages within the large pages and free up memory allocated to such smallpages through page sharing. The techniques to free up memory throughpage sharing are described in detail in U.S. Pat. No. 6,789,156, theentire contents of which are incorporated by reference herein.

FIGS. 2A and 2B are conceptual diagrams that illustrate mappings 200 ofsmall and large pages from a virtual memory space 201 to a physicalmemory space 202. Small page mappings are indicated by a single thickarrow (e.g., arrow 210). It should be understood that each small page invirtual memory space 201 on the left side of this arrow is mapped to acorresponding small page in physical memory space 202 on the right sideof this arrow. A large page mapping is indicated by a thinner arrow(e.g., arrow 220).

When a page in physical memory space 202 is mapped large, page tablesare updated to indicate a large page mapping. For example, in anembodiment that employs hierarchically arranged page tables, the largepage is mapped not from the lowest-level page table but from a pagetable that is next level up from the lowest-level page table. Inaddition, the page table entry corresponding to the large page isupdated to indicate there is no page table at a lower level. FIG. 2Ashows a state of memory 104 where physical memory space 202 ispartitioned into both small and large pages. One of the large pages isshown as LP 221. In situations where it makes sense for the large pageto be broken up, e.g., when there are opportunities for page sharing atthe granularity of small pages within the large page and memory isbecoming scarce, the large page may be broken up into small pages. FIG.2B shows the state of memory after LP 221 has been broken up into smallpages (SP 211).

FIGS. 3A and 3B are conceptual diagrams that illustrate changes in themappings of small pages when page sharing is executed for those smallpages of SP 211 that have the same content as one of shared pages inmemory 104, which may be pages that have known content, such as a zeropage or a page with a simple pattern (represented as page 301 in FIG.3B). Other shared pages are represented as page 302 in FIG. 3B. FIG. 3Ashows the state of memory 104 prior to the execution of page sharing,and FIG. 3B shows the state of memory 104 after the execution of pagesharing. As a result of the page sharing, one or more small pages arefreed up. In this example, six small pages indicated with “F” are shownas being freed up.

FIG. 4 depicts a flow diagram that illustrates a method that includesthe steps of identifying page sharing opportunities, according to anembodiment. In the embodiment illustrated herein, page sharing module150 is rotating through a complete set of large pages on a periodicbasis and executing this method in a background thread. The rate atwhich this method is executed may be adjusted according to the state ofmemory 104. For example, as memory becomes scarce as determined by thememory scheduling module, this rate is increased, and conversely, asmemory becomes more plentiful as determined by the memory schedulingmodule, this rate is decreased. In addition, for efficiency, matchesbetween small pages are found by comparing hashes of the small pages. Inalternative embodiments, other techniques for finding matches betweensmall pages which do not require hash computations may be used.

This method begins at step 403 where page sharing module 150 selects alarge page to be examined for page sharing opportunities. Then, withinthe large page, page sharing module 150 selects a small page at step 405and computes a hash of the small page at step 406. Then, page sharingmodule 150 compares the hash computed at step 406 with hashes of othersmall pages at step 407. If a match is determined (step 409), a countervariable, CNT, is incremented by one (step 411). This comparison iscarried out for all the small pages within the large page selected atstep 403 that are non-zero pages and the counter variable, CNT,incremented accordingly. For small pages that are zero pages, steps 406,407, and 409 are skipped and step 411 is executed after the zero smallpage is detected upon selection at step 405. When it is determined atstep 413 that there are no more small pages in the selected large page,step 415 is executed where the value of the counter variable, CNT, isstored in a table or in any similar data structure and then reset tozero. The table or similar data structure tracks the number of pagesharing opportunities for each of the different large pages of memory104 at a given time. It should be recognized that as pages within memory104, including the large pages being scanned according to this method,are modified as time progresses, content within a particular large pagemay change and, as a result, the number of actual page sharingopportunities may be different from the number stored in the table, whenthis large page is broken up to enable page sharing. When the last largepage has been processed, as determined at step 417, page sharing module150 estimates at step 419 the total amount of memory that could bereclaimed by executing page sharing for shareable small pages within thelarge pages. In one embodiment, page sharing module 150 obtains thisestimate by summing up all of the CNT values stored in the trackingtable and multiplying by the small page size. In another embodiment,page sharing module 150 applies a correction factor to this estimate,where the correction factor is determined by tracking the actual numberof small pages that are determined to be shareable (described below inconjunction with step 518) when a large page is broken up and comparingthe actual number with the CNT value stored for that particular largepage. For example, if page sharing module 150 determines that itoverestimated the number of shareable pages by a factor of two per largepage, the total amount of memory that could be reclaimed would beestimated by dividing by two each of the CNT values stored in thetracking table and then multiplying the sum of all the adjusted CNTvalues by the small page size.

In the embodiment described above, a single CNT value is maintained foreach large page. In alternative embodiments, multiple CNT values may bemaintained for each large page. For example, one CNT value may bemaintained for sharable pages that are zero pages and another CNT valuemay be maintained for non-zero sharable pages.

FIG. 5 depicts a flow diagram that illustrates a method that includesthe steps of breaking up large pages to enable page sharing, accordingto an embodiment. In the embodiment illustrated herein, page sharingmodule 150 is programmed to execute this method in a background threadon a periodic basis.

This method begins at step 502 where the memory scheduling module ofhypervisor 114 assesses the memory state of memory 104. This may be doneby examining memory usage statistics maintained by hypervisor 114. Ifthe amount of free memory is below a predefined minimum as determined atstep 504, additional large page mapping by hypervisor 114 is disabledand a threshold is set for breaking up a large page so that small pageswithin the large page can be freed up through page sharing (step 510).The threshold is a number that represents the minimum number ofshareable small pages within a large page for that large page to bebroken up to enable small pages therein to be freed up through pagesharing. The threshold is adjusted upwards or downwards based on thefree memory state. If the amount of free memory is low, the threshold isadjusted downwards. If the amount of free memory is high, the thresholdis adjusted upwards.

At step 512, page sharing module 150 selects a large page for possiblebreak-up. In one embodiment, the selection is made according to theorder of the large pages appearing in the tracking table. In anotherembodiment, the selection is made according to a descending order of theCNT values with a large page having the largest CNT value beingprocessed first. Further, the selection may take into an accessfrequency of the large page, which may be determined according totechniques described in U.S. patent application Ser. No. 12/834,662, theentire contents of which are incorporated by reference herein. Forexample, for large pages that have an equal number of shareable smallpages, selection preference should be given to the large page whoseaccess frequency is lower (in other words, to large pages that arecolder). In some embodiments, weighting factors are applied to each ofthese two factors for affecting the large page selection. In yet anotherembodiment, the selection of the large page is made randomly. As memorybecomes scarce or as the amount of free memory decreases, a higherweighting factor is given to the number of shareable small shareablepages in a given large page rather than to its coldness value. As memorybecomes more plentiful or as the amount of free memory increases, ahigher weighting factor is given to a given large page's coldness valuerather than to the number of shareable small shareable pages therein.

Then, at step 514, page sharing module 150 determines if the selectedlarge page has at least the requisite number of shareable pages (e.g.,CNT value>threshold, where the CNT value can be the CNT value stored instep 415, or such CNT value that is adjusted by a correction factor asdescribed above). If so, step 516 is executed where page sharing module150 causes the page tables to be updated so that the large page is nowmapped small and page table entries of shareable small pages referenceshared pages. For example, the page table entry corresponding to thelarge page is updated to indicate that there is a page table at a lowerlevel and entries of this page table are updated so that they reference:(i) small pages within the large page that has been broken up (for thosesmall pages that cannot be shared) or (ii) shared pages (for those smallpages that can be shared). It should be understood that a finaldetermination as to whether a small page can be shared or not is made atthis time in accordance with the techniques described in U.S. Pat. No.6,789,156. After this determination for all of the small pages withinthe large page that has been broken up, the actual number of small pagesthat were determined to be shareable is stored in the tracking table atstep 518 so that the correction factor can be determined using thisnumber as described above. Then, at step 520, after pausing for a periodof time, the flow returns to step 502. The amount of time paused may beadjusted downwards if the amount of free memory is less than thepredefined minimum (as determined at step 504) or upwards if the amountof free memory is above the predefined minimum by a sufficient margin(as determined at step 506).

Step 514 is not a part of all embodiments. In embodiments where largepages are ranked based on the selection criteria and/or weightingfactors described above, the large pages will be broken up in orderbased on their ranks until the amount of free memory is no longer lessthan the predefined minimum as determined at step 504. Therefore, insuch embodiments, step 516 will be executed after step 512.

Returning to step 504, if the memory scheduling module of hypervisor 114determines that the amount of free memory is greater than the predefinedminimum, the memory scheduling module of hypervisor 114 performs a checkat step 506 to determine if the amount of free memory is above thepredefined minimum by a sufficient margin (e.g., 1.k times thepredefined minimum, where the constant “k” may be any positive integerand is selected to prevent the free memory state from cycling back andforth between low and high states). Thus, step 508 is executed onlyafter the amount of free memory has increased to above the predefinedminimum by a sufficient margin. At step 508, additional large pagemapping is re-enabled so that hypervisor 114 can return to mapping pageslarge to improve performance.

The various embodiments described herein may employ variouscomputer-implemented operations involving data stored in computersystems. For example, these operations may require physical manipulationof physical quantities—usually, though not necessarily, these quantitiesmay take the form of electrical or magnetic signals, where they orrepresentations of them are capable of being stored, transferred,combined, compared, or otherwise manipulated. Further, suchmanipulations are often referred to in terms, such as producing,identifying, determining, or comparing. Any operations described hereinthat form part of one or more embodiments of the invention may be usefulmachine operations. In addition, one or more embodiments of theinvention also relate to a device or an apparatus for performing theseoperations. The apparatus may be specially constructed for specificrequired purposes, or it may be a general purpose computer selectivelyactivated or configured by a computer program stored in the computer. Inparticular, various general purpose machines may be used with computerprograms written in accordance with the teachings herein, or it may bemore convenient to construct a more specialized apparatus to perform therequired operations.

The various embodiments described herein may be practiced with othercomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers, and the like.

One or more embodiments of the present invention may be implemented asone or more computer programs or as one or more computer program modulesembodied in one or more computer readable media. The term computerreadable medium refers to any data storage device that can store datawhich can thereafter be input to a computer system—computer readablemedia may be based on any existing or subsequently developed technologyfor embodying computer programs in a manner that enables them to be readby a computer. Examples of a computer readable medium include a harddrive, network attached storage (NAS), read-only memory, random-accessmemory (e.g., a flash memory device), a CD (Compact Discs)—CD-ROM, aCD-R, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, andother optical and non-optical data storage devices. The computerreadable medium can also be distributed over a network coupled computersystem so that the computer readable code is stored and executed in adistributed fashion.

Although one or more embodiments of the present invention have beendescribed in some detail for clarity of understanding, it will beapparent that certain changes and modifications may be made within thescope of the claims. Accordingly, the described embodiments are to beconsidered as illustrative and not restrictive, and the scope of theclaims is not to be limited to details given herein, but may be modifiedwithin the scope and equivalents of the claims. In the claims, elementsand/or steps do not imply any particular order of operation, unlessexplicitly stated in the claims.

Virtualization systems in accordance with the various embodiments may beimplemented as hosted embodiments, non-hosted embodiments or asembodiments that tend to blur distinctions between the two, are allenvisioned. Furthermore, various virtualization operations may be whollyor partially implemented in hardware. For example, a hardwareimplementation may employ a look-up table for modification of storageaccess requests to secure non-disk data.

Many variations, modifications, additions, and improvements arepossible, regardless the degree of virtualization. The virtualizationsoftware can therefore include components of a host, console, or guestoperating system that performs virtualization functions. Pluralinstances may be provided for components, operations or structuresdescribed herein as a single instance. Finally, boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of the invention(s). Ingeneral, structures and functionality presented as separate componentsin exemplary configurations may be implemented as a combined structureor component. Similarly, structures and functionality presented as asingle component may be implemented as separate components. These andother variations, modifications, additions, and improvements may fallwithin the scope of the appended claim(s).

We claim:
 1. A method of reclaiming memory in a computer system wherethe memory is partitioned and accessed as small pages and large pages,comprising: selecting a large page that is comprised of a group of smallpages based on a number of small page sharing opportunities identifiedtherein; updating mappings for the memory so that a mapping to theselected large page is changed to mappings to small pages, at least oneof the small pages being a shared small page; and marking one or more ofthe small pages in the group as free.
 2. The method of claim 1, furthercomprising: scanning each of the large pages and determining a number ofsmall pages therein that can be shared, wherein the selecting is basedon the relative number of shareable small pages in the large pages. 3.The method of claim 2, wherein the selecting is based on an accessfrequency of the large pages.
 4. The method of claim 1, furthercomprising: determining the small pages in the group that can beactually shared, wherein the small pages that can be actually shared aremarked as free.
 5. The method of claim 1, wherein the large page isselected if the number of small page therein that can be shared isgreater than a threshold.
 6. The method of claim 5, further comprising:setting the threshold according to memory usage by the computer system,wherein the threshold is set lower as the memory usage increases.
 7. Themethod of claim 1, further comprising: scanning each of the large pagesand determining from the scanning a number of small pages therein thatcan be shared; storing the number of small pages that can be shared in atable, wherein the large page is selected according to the number storedin the table when a memory usage by the computer system exceeds athreshold.
 8. The method of claim 7, further comprising: rescanning eachof the large pages and determining from the rescanning a new number ofsmall pages therein that can be shared; and storing the new number ofsmall pages that can be shared in the table.
 9. The method of claim 8,further comprising: upon said updating, determining an actual number ofsmall pages therein that can be shared; and estimating an amount ofmemory that could be reclaimed by page sharing based on the table andthe actual number.
 10. The method of claim 1, wherein the large page isat least twice the size of the small page.
 11. A method of determiningan amount of memory that can be reclaimed through page sharing in acomputer system where the memory is partitioned and accessed as smallpages and large pages, comprising: scanning each of the large pages anddetermining from the scanning a number of small pages therein that canbe shared; and determining an amount of memory that can be reclaimedbased on a total number of small pages that can be shared as determinedfrom the scanning of all of the large pages.
 12. The method of claim 11,further comprising: rescanning each of the large pages and determiningfrom the rescanning a new number of small pages therein that can beshared; and updating the amount of memory that can be reclaimed based ona total number of small pages that can be shared as determined from therescanning of all of the large pages.
 13. The method of claim 12,further comprising: maintaining a data structure in memory to trackresults of the scanning; periodically scanning the data structure forone or more candidates for page sharing; updating mappings for thememory so that a mapping to each of the candidates is changed tomappings to small pages; determining the small pages that can beactually shared and changing the mappings to the small pages that can beactually shared to shared small pages; and marking the small pages thatcan be actually shared as free.
 14. The method of claim 13, wherein therate of the periodical scanning is set according to memory usage by thecomputer system.
 15. The method of claim 14, wherein the rate is sethigher as the memory usage increases.
 16. The method of claim 13,further comprising: selecting the candidates for page sharing based onfirst and second selection criteria, the first selection criteria beingthe number of pages therein that can be shared and the second selectioncriteria being an access frequency thereof.
 17. The method of claim 16,wherein first and second weighting factors are applied respectively tothe first and second selection criteria.
 18. The method of claim 17,wherein the first weighting factor increases as the memory usageincreases and the second weighting factor decreases as the memory usageincreases.
 19. A computer system comprising: virtual machines executedtherein; and a hypervisor configured to support execution of the virtualmachines, wherein the hypervisor is configured to map a virtual memoryspace to a physical memory space that is logically partitioned intolarge pages and small pages, and to determine a number of small pageswithin a given large page whose contents are the same as another smallpage.
 20. The computer system of claim 19, wherein the contents of theanother small page are zeros.